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INTRODUCTION 


The  goal  of  a  precision  strike  is  to  transit  to  the  theater  safely  and  to  project 
lethal  force  precisely  against  a  sophisticated  enemy,  all  with  relatively  few  casualties 
and  a  minimum  of  collateral  damage.  The  ability  to  attack  safely  and  precisely  in  all 
lighting  and  weather  conditions  requires  precision  aimpoint  selection  either  at  stand¬ 
off  ranges  or  during  single-pass,  low  level,  direct  attacks.  For  stand-off  attacks,  new' 
targeting  concepts  are  needed  in  order  to  utilize  imaging  sensor  information  that  has 
never  before  been  available  in  the  cockpit  of  an  attack  aircraft.  For  successful  low 
level  attacks,  new  techniques  must  be  developed  that  permit  the  pilot  to  locate  a  target 
and  release  a  weapon  in  the  few  seconds  available  in  a  single,  high  speed  pass. 

The  Land-Attack  Multi-sensor  Correlation  (LMC)  project  under  development  at 
the  Naval  Air  Warfare  Center  Weapons  Division  (NAWCWPNS),  China  Lake,  Calif., 
is  designed  for  either  stand-off  or  direct  attack.  It  will  locate  offset  aim  points  or 
targets  by  correlating  incoming  sensor  imagery  in  a  cockpit  with  reference  materials 
derived  from  three-dimensional  models  of  the  target  or  aim  point  area  stored  on  board 
the  attack  aircraft.  These  three-dimensional  models  are  created  prior  to  the  mission 
from  various  sources  (including  satellite  imagery,  aerial  surveillance  photographs, 
and  digital  maps).  During  the  mission,  the  position  (azimuth  and  elevation)  and 
orientation  of  the  aircraft  sensor  are  used  in  conjunction  with  the  model  to  generate  a 
line  drawing  of  the  target  area  in  the  proper  perspective.  In  the  cockpit  during  the 
mission,  the  LMC  correlates  these  line  drawings  with  edge  enhanced  sensor  imagery 
and  notifies  the  pilot  when  a  peak  correlation  is  found.  The  pilot  need  only  glance  at 
an  overlay  of  the  line  drawing  and  sensor  imagery  to  verify  the  match  and  initiate 
weapon  release.  The  correlation  and  the  pilot  verification  process  can  both  be 
accomplished  in  only  a  few  seconds. 

An  experiment  conducted  in  1992  determined  that  it  took  an  average  of  2.5  to  5 
seconds  for  an  operator  to  determine  if  there  was  a  match  between  a  line  drawing  and 
a  photograph  of  a  scene  (Reference  1).  Various  response  times  (RTs)  in  this  range 
were  associated  with  whether  the  line  drawing  was  overlaid  or  shown  beside  the 
photograph;  whether  it  was  accurately  overlaid,  rotated,  or  offset  from  the 
photograph;  whether  the  color  of  the  line  drawing  was  green  or  white;  and  whether 
the  line  drawing  matched  the  photograph.  Most  importantly,  the  experiment  showed 
that  the  entire  range  of  RTs  of  an  operator  using  the  LMC  was  less  than  the  8  to  10 
seconds  typically  available  for  targeting  in  a  low  level  attack  mission. 

The  experiment  lacked  generality  in  two  ways:  (1)  The  line  drawings  were 
generated  in  a  manner  that  modeled  the  procedures  of  the  LMC,  but  did  not  result  in 
drawings  that  looked  like  the  LMC  drawings.  The  experiment's  drawings  appeared 
more  complete,  with  longer  and  more  connected  lines  than  the  drawings  generated  by 
the  LMC  algorithms.  (2)  The  line  drawings  in  the  experiment  contained  many  more 
pixels  than  the  drawings  used  in  real  time  by  the  LMC  correlator.  New  developments 
had  shown  that  the  number  of  pixels  that  the  correlator  could  handle  in  real  time  was 
somewhere  between  1500  and  2500;  many  of  the  line  drawings  in  the  experiment 
exceeded  10,000  pixels. 
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Intuitively,  some  templates  and  images  are  probably  easier  to  match  than  others; 
also  intuitively,  a  large  reduction  in  the  number  of  pixels  that  constitute  a  line 
drawing  should  make  pattern  matching  more  difficult.  However  a  search  of  the 
literature  did  not  provide  consistent  or  relevant  information  on  the  accuracy  and  speed 
of  pattern  matching  of  this  type.  Most  of  the  experiments  that  are  reported  do  not  deal 
with  pattern  matching  but  rather  with  pattern  recognition  (the  ability  to  assess  with 
accuracy  if  one  has  or  has  not  previously  seen  a  given  pattern)  or  with  pattern  recall 
(the  subsequent  ability  to  provide  an  accurate  description  of  a  pattern  that  one  has 
seen  earlier,  without  being  provided  with  an  example).  Thus  application  of  the 
experimental  results  to  pattern  matching  is  not  necessarily  straightforward  In 
addition,  the  results  from  experiment  to  experiment  often  conflicted,  and  different 
specific  measures  for  recall  and  recognition  were  used  from  report  to  report. 

For  example,  Ryan  and  Schwartz  found  that  line  drawings  of  objects  were  the 
most  difficult  to  perceive  accurately,  and  that  cartoons  were  the  easiest,  with 
photographs  and  shaded  drawings  about  equal  to  each  other  and  falling  between  the 
other  two  (Reference  2).  The  perception  measure  taken  in  this  experiment  was 
exposure  time  of  the  image  until  an  accurate  pattern  matching  response  was  produced 
by  the  observer.  Pattern  matching  in  this  case  involved  manipulating  an  object  (such 
as  a  set  of  switches)  so  that  the  object  appeared  in  the  same  configuration  and  aspect 
as  the  objects  in  the  pictures.  There  was  a  substantial  interaction  between  the  object 
being  portrayed  and  the  optimum  method  of  portrayal,  so  that  for  some  objects,  line 
drawings  led  to  the  worst  pattern  matching,  but  for  others,  line  drawings  worked  very 
well. 

In  contrast,  Nelson,  Metzler,  and  Reed  found  that  there  were  no  differences 
between  photographs,  unembellished  line  drawings  (drawings  without  details),  and 
embellished  line  drawings  (with  added  details)  for  either  immediate  or  delayed 
recognition  (Reference  3).  Recognition  was  measured  as  the  percent  of  trials  in  which 
the  subjects  correctly  chose  the  images  that  they  had  previously  seen  in  two 
alternative  forced  choice  tests.  All  three  were  significantly  better  than  verbal 
descriptions  for  recognition  of  the  images. 

A  final  example  by  Loftus  and  Bell  (using  Nelson's  photographs  and  line 
drawings)  showed  that  the  photographs  were  more  frequently  recognized  in  forced 
choice  tasks  than  were  line  drawings  (Reference  4).  Further,  all  imagery  in  which  a 
detail  was  recalled  was  recognized  more  readily  than  imagery  with  no  recall  of 
details.  Loftus  and  Bell  believe  that  recognition  of  imagery  may  be  based  either  on 
specific  detail  information  or  on  general  visual  information.  The  latter  process  accrues 
information  gradually  over  time,  explaining  the  common  finding  that  the  longer  an 
image  is  viewed,  the  more  information  is  accrued.  The  authors  theorize  that  while 
accruing  information,  the  observer  is  simultaneously  engaged  in  a  search  for  a 
potentially  informative  detail.  If  such  a  detail  is  located,  recognizability  of  that  image 
makes  a  quantum  jump.  The  potential  of  finding  a  significant  detail,  they  believe,  is 
greater  in  a  photograph  than  in  a  line  drawing,  even  if  the  line  drawing  is  embellished. 

Historically,  recognition  of  imagery  has  been  believed  to  be  based  upon  two 
types  of  information:  a  verbal  component  (many  experiments  have  shown  that  verbal 
descriptions  of  a  scene  being  viewed  improves  subsequent  recognition),  and  a 
nonverbal  or  visual  component  (scene  recognition  still  occurs  when  verbalization  is 
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prevented).  The  authors  propose  that  the  finding  of  a  significant  detail  acts  much  the 
same  way  that  verbalizations  aid  in  image  recognition.  That  is,  what  they  "...  have 
termed  specific  detail  information  is  equivalent  to  what  others  have  described  as  the 
verbal  component  of  picture  memory  ..."  (Reference  4,  page  1 12). 

The  findings  reported  in  the  literature  and  summarized  above,  and  their 
interpretations,  may  also  apply  to  the  pattern  matching  required  of  the  operator  who 
verifies  the  output  of  the  LMC.  While  the  experiments  do  not  provide  consistent 
results,  the  literature  suggests  that  recognition  of  objects  is  possible,  if  not  optimum, 
from  line  drawings  of  the  objects.  Presumably  also,  line  drawings  can  be  used  for 
pattern  matching.  Similarly,  if  recognition  is  based  both  on  visual  information  and  on 
a  verbal/detail  component,  pattern  matching  is  also  probably  based  upon  similar 
components.  Therefore  pattern  matching  will  probably  be  facilitated  by  images  and 
templates  that  provide  not  only  general  visual  information  but  also  details  that  can  be 
matched.  If  that  is  the  case,  the  fewer  the  pixels  in  a  line  drawing,  the  less  likely  it  is 
that  a  detail  will  be  included,  and  the  more  difficult  it  should  be  to  verify  a  match.  By 
similar  reasoning,  verbal  descriptions  of  scenes  and  target  areas  should  also  facilitate 
pattern  matching. 

This  report  describes  the  results  of  two  experiments  that  re-examine  operators' 
accuracies  and  RTs  to  recognize  matches  and  mismatches  between  line  drawings  and 
photographs.  The  line  drawings  used  in  the  experiments  were  generated  by  the 
procedures  used  in  the  LMC.  The  experiments  varied  the  number  of  pixels  in  the  line 
drawings,  the  manner  in  which  the  pixels  were  varied,  and  the  kind  of  pre-mission 
experience  the  operators  had  with  the  imagery.  In  addition,  the  performance  results 
were  used  to  demonstrate  an  approach  that  might  be  used  to  set  a  criterion  score  for 
calling  out  a  match  by  the  LMC  correlator. 

METHOD 

Two  experiments  were  designed  together  and  share  the  same  materials.  Seventy- 
two  photographs  of  a  variety  of  scenes  were  digitized  for  computer  processing.  Each 
of  the  images  was  processed  using  the  LMC  algorithms  so  that  after  processing,  edges 
of  sufficient  contrast,  continuity,  and  length  in  the  original  picture  were  represented 
by  lines.  The  lines  for  each  picture  were  made  of  approximately  5000  pixels;  these 
constituted  the  basic  line  drawings  from  which  the  stimuli  for  both  experiments  were 
derived. 


EXPERIMENT  1 


Stimuli 

For  each  of  the  72  photographs  in  the  first  experiment,  12  line  drawings  were 
created.  Each  of  the  12  line  drawings  represented  a  variation  of  level  in  two  basic 
factors:  Pixel  Count,  which  varied  by  having  either  1000,  2000,  3000,  or  4000  pixels 
in  a  drawing;  and  by  Reduction  Method,  which  was  used  to  reduce  the  basic  line 
drawings  from  5000  pixels  to  the  required  pixel  count.  In  this  experiment  all 
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Reduction  Methods  were  computerized  processes.  There  were  three  Reduction 
Method  levels: 

1.  The  Long  Reduction  Method  ("Long"),  in  which  the  computer  successively 
removed  the  shortest  line  from  the  basic  drawing  until  the  appropriate  total  number  of 
pixels  remained.  Thus  the  final  line  drawings  in  "Long”  condition  were  made  up  of 
the  longest  lines  from  the  basic  drawing;  there  were  four  pixel  count  levels  in  the 
Long  condition,  which  contained  either  1000,  2000,  3000,  or  4000  pixels  (see  Figure 
D- 


2.  The  Center  Reduction  Method  ("Center"),  in  which  the  computer  successively 
removed  lines,  starting  with  the  farthest  away  from  the  physical  center  of  the  basic 
drawing,  until  the  appropriate  number  of  pixels  remained.  Thus  the  final  "Center" 
drawings  were  composed  primarily  of  lines  in  the  physical  center  of  the  drawing  and 
contained  1000,  20()0,  3000,  or  4000  pixels. 

3.  The  Random  Reduction  Method  ("Random"),  in  which  the  computer 
successively  removed  random  lines  from  the  basic  line  drawing  until  1000,  2000, 
3000,  or  4000  pixels  remained.  This  was  essentially  a  control  condition.  Figures  1  and 
2  shows  examples  of  the  three  Reduction  Methods. 


(a)  "Oil  tanks  on  the  hills  by  the  shore." 

FIGURE  1 .  An  Example  of  One  of  the  72  Photographs  Used  in  the  Experiment, 
and  the  Line  Drawings  With  1000,  2000,  3000,  and  4000  Pixels  Generated 
For  It  by  the  LMC  Using  the  "Long"  Reduction  Method. 
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(b)  Basic  line  drawing  used  in  (c)  1000  pixels, 

both  experiments;  5000  pixels. 


FIGURE  l.(Contd.) 
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FIGURE  2.  Oil  Tank  Farm  Line  Drawings  Re¬ 
sulting  From  the  Three  Reduction  Methods. 
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Matches  and  Mismatches 

To  create  the  matching  condition,  the  appropriate  line  drawing  was  superimposed 
over  the  picture  from  which  the  drawing  was  originally  created.  Mismatches  were 
created  by  superimposing  a  line  drawing  over  an  image  from  which  it  was  not 
created.  For  Txismatches,  the  placement  of  the  incorrect  drawing  was  done  with  care, 
to  make  the  "best  matched"  mismatch,  that  is  to  have  the  drawing  lines  correspond  as 
closely  as  possible  to  the  edges  in  the  photograph. 

Equipment 

The  photographs  and  overlaid  line  drawings  w’ere  recorded  using  a  Panasonic 
TQ-2023  Optical  Disk  Recorder.  The  imagery  was  stabilized  during  the  recoding  with 
a  JVC  KM-F250  Frame  Synchronizer.  During  playback  both  the  Panasonic  Recorder 
and  a  Panasonic  TQ-2027  Optical  Disk  Player  were  used  to  present  the  imagery  on  a 
Sony  PVM-1910  Trinitron  monitor.  The  experiment  was  controlled  and  the  results 
recorded  by  a  Foundation  80386  Computer. 

Subjects 

Experiment  i  used  16  subjects  who  were  male  or  female  civilian  employees  of 
NAWCWPNS,  China  Lake.  No  individual  subject  participated  in  more  than  one  test 
connected  with  any  part  of  these  experiments. 

PROCEDURE 

Experimental  Design 

Experiment  1  used  a  factorial  design  with  three  fully  crossed  factors.  The  factors 
were;  (1)  Pixel  Count  (1000,  2000,  3000,  4000);  (2)  Reduction  Method  (Long, 
Center,  and  Random);  and  (3)  Matchi  (Matched  and  Mismatched). 

Randomization  Procedure 

Prior  to  recording  any  imagery  on  the  laser  disc,  a  set  of  image-line  drawing 
pairs  was  established  for  each  person.  For  matches,  each  subject  had  the  same  set:  a 
line  drawing  superimposed  over  the  photograph  from  which  it  was  created.  For 
mismatches,  each  subject  saw  the  same  72  photographs,  but  each  was  paired  with  a 
mismatched  overlay  created  from  one  of  the  71  other  photographs.  The  ba.se 
photograph  for  a  mismatch  line  drawing  was  preselected  randomly,  without 
replacement,  from  the  pool  of  possible  mismatches.  Thus  there  were  72  matching 
pairs  and  72  mismatching  ones,  for  a  total  of  144  images. 

Next,  a  matrix  was  built  that,  for  each  of  16  subjects,  assigned  the  72  matches 
and  72  mismatches  to  each  experimental  condition.  The  design  of  the  experiment 
called  for  6  replications  of  the  12  conditions  (3  Reduction  Method  conditions  x  4 
pixel  count  conditions)  for  matches  and  for  6  replications  of  the  12  for  mismatches. 
Thus  altogether  there  were  72  matching  trials  and  72  mismatching  trials.  For  each 
subject  separately,  the  72  matched  and  72  mi.smatched  pairs  described  above  were 
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randomly  assigned,  without  replacement,  to  these  72  matching  and  mismatching 
trials. 

Test  Procedure 

Subjects  were  told  the  general  nature  of  the  experiment  and  shown  the  equipment 
used  in  the  test.  Then  the  computer  provided  instructions  for  subjects  on  what  they 
would  see,  provided  three  examples  of  matches  and  mismatches,  and  instructed  th^m 
on  how  they  should  respond.  In  each  trial  the  subject  was  to  determine  as  rapidly  as 
possible  whether  the  line  drawing  matched  or  mismatched  the  photograph.  Each 
subject  was  then  shown,  in  random  order,  the  144  images  that  had  previously  been 
recorded.  Responses  and  RTs  were  recorded  for  each  trial.  The  entire  test  took  from 
1 5  and  30  minutes  for  each  subject. 

RESULTS 

Percentage  of  Correct  Responses 

The  mean  percentage  of  correct  responses  for  the  three  Reduction  Methods,  the 
four  levels  of  pixel  counts,  and  the  two  matching  conditions  are  listed  in  Table  1 . 

TABLE  I.  Percentage  of  Correct  Responses  in  Experiment  1. 


Correct  response.  %  Response  time,  sec 


Reduction  Method  1 
Random 

85.4 

3.25 

Center 

82.1 

3.21 

Long 

92.5 

2.75 

Pixel  limit 

1000 

82.5 

3.2 

2000 

86.3 

3.18 

3000 

89.1 

2.96 

4000 

88.7 

2.95 

Matching 

Matched 

86.7 

3.14 

Mismatched 

87.5 

3.01 

These  findings  were  analyzed  using  a  repeated  measures  Analysis  of  Variance 
(ANOVA).  The  main  effect  for  Reduction  Method  was  significant  [F  (2,  28)=  17.22. 
P<  0.001],  as  was  the  main  effect  for  Pixel  Count  [F  (3,  42)=  5.41,  P<  0.003].  There 
were  no  significant  differences  in  percent  correct  between  matches  and  mismatches, 
with  matches  recognized  correctly  86.7%  of  the  time,  and  mismatches  recognized 
correctly  87.5%  of  the  time.  The  2  two-way  interactions  that  involved  matching  were 
both  significant,  with  Matching  x  Reduction  Method  significant  at  the  0.004  level  [F 
(2.28)  =  6.71]  and  Matching  x  Pixel  Count  significant  at  less  than  the  0.001  level 
[F(3,  42)  =  8.20].  These  relationships  are  shown  in  Figures  3  and  4. 
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Random  Cantor  Lena 


Mlamateliad 


Matched 


Matching 


FIGURE  3.  Percentage  of  Correct  Responses  for  Reduction  X  Matching  Interaction. 


1000  2000  3000  4000 

Pixel  Count 

'  FIGURE  4.  Percentage  of  Correct  Responses  for  Matching  x  Pixel  Count  Interaction. 
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Contrasts  for  the  Reduction  x  Matching  Interaction  show  that  for  percent  correct. 
Long  in  the  matching  condition  was  significantly  higher  than  for  all  of  the  other  five 
conditions  [F(l,14)=  7.3,  P<  0.02],  and  that  the  other  five  conditions  did  not  vary 
from  one  another.  Analysis  of  the  Pixel  Count  x  Matching  interaction  show  that  the 
percentage  of  correct  responses  increased  significantly  with  increases  in  pixel  count 
in  the  mismatches  [F(2,177)=  13.924,  P<0.()bl],  but  not  in  the  matches.  Both  linear 
and  quadratic  components  were  significant  for  the  mismatches,  indicating  the 
presence  of  an  upper  boundary'  in  percent  correct  somewhere  between  3000  and  4000 
pixels. 

The  two  way  interaction  between  Reduction  Method  and  Pixel  Count  w'as 
marginally  significant  [F(6,  84)=  1.15,  P<0.07];  the  general  tendency  was  for  the 
differences  in  percentage  of  correct  responses  for  each  Reduction  Method  became 
smaller  as  the  number  of  pixels  in  the  line  drawing  increased.  This  relationship  is 
shown  in  Figure  5.  There  were  no  other  significant  findings  in  the  analysis;  the  three 
way  interaction  was  not  significant. 


FIGURE  5.  Percentage  of  Correct  Respon.ses  for  Reduction  x  Pixel  Count  Interaction. 


Response  Time 

The  ANOVA  for  RT  used  the  mean  RTs  for  correct  respon.ses  only  (see  Table  I ). 
The  only  significant  main  effect  in  the  analysis  was  for  Reduction  Method;  contrasts 
showed  that  the  Long  Reduction  Method  led  to  significantly  faster  responses  than 
both  the  Center  and  Random  methods.  Center  and  Random  Reduction  Methods  did 
not  differ  from  each  other  in  RT. 
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There  was  a  significant  interaction  between  Matching  and  Reduction  Method 
[F(2,28)  =  12.33,  P  <  0.001]  and  a  marginally  significant  interaction  between 
Matching  and  Pixel  Count  [F(3,42)  =  2.58,  P  <  0.07].  These  interactions  are  shown  in 
Figures  6  and  7.  Contrasts  analyzing  the  Matching  x  Reduction  M:Mhod  interaction 
show  that  the  Long  for  matches  had  a  significantly  faster  RT  than  the  other  five 
conditions  [F(l,14)=  13.27,  P  <  0.003].  In  the  Matching  X  Pixel  Count  interaction, 
most  of  the  variation  in  RT  seems  to  be  a  reduction  in  RT  as  pixel  count  increases  in 
the  mismatches,  but  there  was  little  variation  in  RT  to  recognize  matches. 


FIGURE  6.  Response  Times  for  Matching  x  Reduction  Method  Interaction. 
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DISCUSSION 

Overall,  the  Long  line  drawings  were  superior  in  both  accuracy  and  speed  to  the 
drawings  produced  by  the  other  Reduction  Methods.  The  advantages  of  the  Long  line 
drawings  were  apparent  primarily  in  the  matching  conditions,  where  operators  using 
them  could  verify  matches  approximately  I  second  faster  and  with  greater  than  1 09c 
higher  accuracy  than  they  could  using  line  drawings  made  with  the  Focus  or  Random 
Reduction  Methods  (see  Figure  2).  In  the  final  phases  of  a  low  level  attack,  1  second 
saved  translates  into  approximately  1/8  mile,  and  represents  a  large  proportion  of  the 
8  to  10  seconds  available  for  targeting. 

The  findings  for  Pixel  Count  showed  that  in  general,  operators  perform  faster  and 
more  accurately  when  there  are  more  pixels  in  a  line  drawing.  The  increases  in 
performance  accuracy,  however,  level  off  between  3000  and  35(X)  pixels,  and  beyond 
that,  there  is  no  further  performance  gain.  Further,  the  effect  of  pixel  count  shows 
primarily  in  the  operators’  ability  to  recognize  mismatches,  which  in  the  reai  LMC 
operation,  would  equate  to  the  operators'  ability  to  correct  false  alarms  that  may  be 
called  out  by  the  correlator.  The  pixel  count  had  far  less  effect  on  the  operators’ 
ability  to  verify  matches. 

The  design  implications  from  Experiment  1  are  clear; 

1 .  Make  templates  that  consist  of  fewer,  longer  lines 

2.  If  the  final  implementation  of  the  LMC  produces  a  large  number  of  false 
alarms  that  need  to  be  corrected  by  an  operator,  then  increase  the  number  of 
pixels  in  the  line  drawings  to  3500  in  order  to  increase  the  ability  to  recognize 
the  false  alarms. 

These  two  steps  should  maximize  the  contribution  of  the  operator  in  system 
performance  of  the  LMC. 


EXPERIMENT  2 


INTRODUCTION 

While  the  first  experiment  demonstrated  that  Long  line  drawings  produced 
superior  pattern  matching  performance,  it  is  clear  that  some  of  the  lines  used  in  these 
drawings  are  surprising  and  would  not  have  been  included  if  the  drawings  had  been 
made  by  hand  instead  of  by  computer.  While  all  of  the  lines  in  the  drawings 
corresponded  to  an  edge  in  the  photograph  from  which  it  was  made,  many  edges  were 
omitted  that  were  more  important  from  a  conceptual  point  of  view,  but  which  had  less 
contrast.  We  could  not  help  wondering  whether  operator  verification  of  pattern 
matches  of  the  sort  required  by  the  LMC  might  be  fastest  and  most  accurate  if  the  line 
drawing  used  for  matching  represented  the  scene  in  a  way  more  cognitively 
compatible  with  operators’  expectations.  We  examined  this  possibility  in  Experiment 
2. 


The  first  experiment  failed  to  show  any  advantages  for  the  Center  line  drawings. 
That  result  was  somewhat  surprising  because  the  Center  method  of  reduction  seemed 
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to  emphasize  the  central  area  of  a  scene,  which  would  be  likely  to  contain  the  greatest 
amount  or  the  most  important  information.  However,  further  e.xamination  of  this  idea 
led  to  the  thought  that  the  area  of  greatest  importance  in  a  scene  might  not  be  in  the 
physical  center  of  the  picture,  but  might  well  be  in  some  other,  off-center  area.  So  in 
Experiment  2  we  reexamined  this  method  of  reduction  so  that  the  line  drawings 
would  focus  on  an  area  of  high  information  content  for  each  picture,  wherever  it  may 
be  in  the  picture. 

Finally  in  Experiment  2,  we  wanted  to  see  if  the  operator's  participation  in 
making  a  line  drawing  or  in  careful  study  of  a  scene  would  enhance  his  subsequent 
performance  in  verifying  the  output  of  the  LMC.  Would  pilots  benefit  from  making 
their  own  line  drawings  in  mission  planning,  or  would  they  add  accuracy  and  speed  in 
using  the  LMC  by  adding  verbal  commentary  to  imagery  obtained  in  the  planning 
process? 

METHOD 

Stimuli 

The  line  drawings  in  the  second  experiment  were  varied  only  by  the  Reduction 
Method,  which  had  four  levels: 

1.  The  Concept  Reduction  Method  ("Concept"),  which,  unlike  the  methods  of 
Experiment  1,  was  done  manually.  First,  brief  two-line  written  descriptions  of  each  of 
the  72  images  were  given  by  four  volunteers.  These  descriptions  were  collated  and 
from  them,  descriptions  were  generated  that  represented  a  consensus  description.  The 
basic  line  drawings  were  then  manually  reduced  so  that  only  lines  contained  in  the 
consensus  descriptions  remained.  For  example,  the  consensus  description  of 
photograph  number  one  (see  Figure  1)  was  "Oil  tanks  on  the  hills  by  the  shore."  Only 
lines  representing  hills,  shore,  or  oil  tanks  were  left  in  the  line  drawings.  Lines 
representing  roads,  buildings,  or  docks  were  removed.  (Figure  8  provides  examples  of 
the  line  drawings  constructed  by  the  Reduction  Methods  in  Experiment  2.) 

2.  The  Focus  Method  ("Focus")  was  similar  to  the  Center  Reduction  Method 
described  for  Experiment  1,  except  that  lines  were  successively  removed  that  were  the 
furthest  away  from  a  focal  point  of  interest  in  the  photograph  (rather  than  from  the 
physical  center).  The  focal  point  of  interest  for  each  picture  was  determined  manually 
following  a  procedure  outlined  by  Mackworth  and  Morandi,  who  showed  that  their 
manner  of  eliciting  subjective  judgments  of  picture  areas  of  importance  correlated 
positively  with  the  total  number  of  visual  fixations  on  those  areas  (Reference  5). 
Focal  points  were  determined  by  having  seven  judges  place  a  circle  around  what  they 
each  thought  was  the  focal  point  of  the  picture.  The  focal  point  for  each  picture  was 
determined  to  be  the  one  chosen  by  the  largest  number  of  judges,  and  lines  furthest 
from  this  point  were  successively  removed. 

3.  The  Long  Reduction  Method  ("Long"),  which  was  described  in  the  "Stimuli ' 
section  of  Experiment  1 . 

4.  The  Random  Reduction  Method  ("Random"),  which  was  de.scribed  in  the 
"Stimuli"  section  of  Experiment  1. 
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The  pixel  count  of  the  four  line  drawings  made  for  each  photograph  in  Experiment  2 
was  held  constant  by  matching  each  one  to  the  pixel  count  of  the  line  drawing 
generated  by  the  Concept  Reduction  Method.  The  pixel  counts  for  the  Concept 
drawings  were  kept  within  the  limits  imposed  by  the  requirements  for  real  time 
correlation  by  the  LMC;  the  counts  ranged  from  712  to  2668  pixels  in  the  72 
drawings.  In  the  four  line  drawings  produced  for  each  photograph,  the  pixel  count  dsd 
not  vary  more  than  60  pixels.  (The  Appendix  to  this  report  lists  the  verbal 
de.scriptions  of  each  photograph  and  the  pixel  count  for  the  Concept  Reduction 
Method.) 


FIGURE  8.  The  Oil  Tank  Farm  Line  Drawings  Generated  by  the  Reduction  Methods 

Used  in  Experiment  2. 


I 
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Preview  Conditions 

In  Experiment  2,  subjects  were  given  three  different  pretrial  exposures  with  the 
photographs.  In  the  Verbal  Condition,  subjects  were  given  20  seconds  to  write  short 
descriptions  of  one  third  of  the  photographs  that  were  presented  in  hard  copy.  In  the 
Image  Condition,  subjects  were  given  hard  copy  photographs  of  a  second  third  of  the 
scenes,  each  covered  by  a  transparency.  They  were  given  20  seconds  to  trace 
significant  features  of  each  photograph  onto  a  transparency,  creating  their  own  line 
drawing  of  each  photograph.  For  the  final  third  of  the  photographs,  subjects  had  no 
pretrial  experience. 

Matches  and  Mismatches 

Matches  and  mismatches  were  created  in  this  experiment  in  the  same  way  that 
they  were  created  in  Experiment  1 . 

Equipment 

The  equipment  used  in  Experiment  1  was  used  in  this  experiment  also. 

Subjects 

As  in  the  first  experiment,  16  China  Lake  employees  were  used  in  this 
experiment.  Subjects  for  Experiment  2  had  not  participated  in  the  fn.^t  experiment, 
pretesting,  or  as  a  volunteer  in  describing  scenes  or  locating  foca!  points  for  this 
experiment. 

PROCEDURE 

Experimental  Design 

In  Experiment  2  the  independent  variables  in  the  factorial  design  were:  fl) 
Preview  Condition  with  three  levels  (None,  Verbal,  and  Image);  (2)  Reduction 
Method  with  four  levels  (Concept,  Long,  Focus,  and  Random):  and  (3)  Matching  with 
two  levels  (Matched  and  Mismatched).  With  six  replications  for  each  condition,  there 
were  144  trials  (72  matching  and  72  mismatching). 

Randomization  Procedure 

The  procedure  described  for  Experiment  1  was  used  to  make  72  matching  and  72 
mismatching  pairs.  In  addition,  a  similar  procedure  was  used  to  assign  tfie  pairs  to  the 
experimental  conditions,  except  that  to  save  time  and  reduce  the  number  of  recordings 
that  had  to  be  made,  the  matrix  was  made  for  only  eight  subjects,  and  the  assignments 
were  repeated  for  the  second  group  of  eight  subjects  that  were  to  participate  in  the 
experiment.  As  in  the  earlier  experiment,  the  72  matching  pairs  were  randomly 
assigned  without  replacement  to  the  72  matching  trials,  and  the  72  mismatching  pairs 
were  similarly  assigned  to  the  72  mismatching  trials.  Once  the.se  assignments  were 
made  for  each  of  the  eight  subjects,  the  appropriate  images  were  constructed  and 
recorded. 
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Test  Procedure 

The  test  procedure  for  Experiment  2  was  similar  to  that  for  Experiment  1 ,  except 
that  prior  to  the  events  outlined  above,  each  subject  first  drew  line  drawings  for  24 
photographs  and  then  wrote  descriptions  of  24  other  photographs.  The  entire  test  for 
each  subject  took  approximately  90  minutes. 

RESULTS 

Percentage  of  Correct  Responses 

Table  2  lists  the  percentage  of  correct  responses  for  each  of  the  three  factors  in 
this  experiment:  Reduction  Method,  Matching  Condition,  and  Preview. 


TABLE  2.  Percentage  of  Correct  Responses  in  Experiment  2. 


Correct  response,  %  Response  time,  sec 


Reduction  Method 

Concept 

93.9 

4.07 

Long 

94.4 

4.33 

Focus 

83.7 

4.94 

Random 

85.3 

5.17 

Preview 

None 

88.7 

4.72 

Template 

90.0 

4.45 

Verbal 

89.3 

4.73 

Matched 

88.6 

4.90 

Mismatched 

90.1 

4.36 

A  within  subjects  ANOVA  on  these  data  showed  that  there  was  a  significant 
main  effect  for  Reduction  Method  [F  (3,51)=18.30,  P<0.001].  Contrasts  show  that  the 
percentage  of  correct  responses  for  the  Focus  and  Random  methods  considered 
together  was  significantly  less  than  for  the  Concept  and  Long  methods  taken  together 
[F  (1,17)-  42.95  P<  0.001],  but  that  there  is  no  difference  in  percentage  of  correct 
responses  between  the  Focus  and  Random  technique  or  between  the  Long  and 
Concept  technique.  No  other  main  effect  was  significant. 

There  was  only  one  significant  interaction.  Matching  x  Reduction  Method  [F{3, 
5I)=  P<0.001].  This  relationship  is  shown  in  Figure  9.  Contrasts  demonstrate  that 
there  are  no  differences  in  percentages  of  correct  responses  in  the  mismatches  due  to 
Reduction  Method,  but  there  are  significant  differences  in  the  matches.  The  Focus  and 
Random  methods  taken  as  a  pair  ted  to  significantly  less  accurate  responses  than  the 
Concept  and  Long  paired  together  [F(l,  17)=  95.78,  P  <  0.001].  However  there  were 
no  differences  between  the  members  of  each  pair  taken  separately. 
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REDUCTION  METHOD 


FIGURE  9.  Percentage  of  Correct  Responses  for  Matching  X  Reduction 

Method  Interaction. 


Response  Time 

Times  for  correct  responses  for  each  of  the  three  factors  in  Experiment  2  are 
shown  in  Table  2.  There  was  only  one  significant  main  effect  in  the  repeated 
measures  ANOVA  used  to  analyze  these  data.  Reduction  Method  [F(3,48)  =  8.75,  P< 
0.001].  Contrasts  showed  that  the  Concept  and  Long  Reduction  Methods  led  to  faster 
RTs  than  the  Focus  and  Random  methods  IF(1,16)=  16.69,  p<0.009].  Further, 
contrasts  also  showed  that  there  were  no  differences  in  RT  between  the  Concept  and 
Long  methods. 

Match  X  Reduction  Method  was  also  significant  [F(3,48)=  7.59,  P<  0.001],  and 
Match  X  Preview  was  marginally  significant  [F(2.32)=  3.17,P<0.06]  (see  Figure  10). 
As  before,  contrasts  showed  that  the  differences  in  Reduction  Methods  showed  only 
in  the  matching  conditions  and  not  in  the  mismatching  conditions,  and  that  the 
Concept  and  Long  methods  led  to  significantly  faster  RTs  than  the  Focus  and 
Random  methods  [F(I,16)=  18.56,  P<0.001].  Contrasts  to  assess  the  significance  of 
the  Preview  x  Match  interaction  showed  that  there  was  a  significant  difference  that 
showed  only  in  the  matched  condition,  such  that  pretraining  in  making  line  drawings 
led  to  significantly  faster  RTs  than  either  of  the  other  two  pretesting  conditions  (no 
preview  or  verbal  descriptions),  F(l,  16)  =  6.76  and  7.46,  respectively,  P  <0.02  and 
<0.01]. 
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REDUCTION  METHOD 


FIGURE  10.  Response  Times  for  Matching  X  Reduction  Method  Interaction. 


Finally,  the  three  way  interaction  of  Match  X  Preview  Condition  X  Reduction 
Method  was  significant  [F(6,96)=2.37,  P<0.04].  This  interaction  is  shown  in  Figure 
11. 


REDUCTION  METHOD  REDUCTION  METHOD 

(a)  Matches.  (b)  Mismatches. 


FIGURE  1 1.  Response  Times  for  Three-Way  Interaction. 
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DISCUSSION 

This  experiment  demonstrates  that  pattern  matching  using  the  Long  and  Concept 
line  drawings  was  both  more  accurate  and  faster  than  pattern  matching  with  the  other 
types  of  line  drawings.  The  magnitude  of  the  improvement  between  the  best  and  the 
worst  was  similar  to  that  found  in  Experiment  1 :  a  savings  of  approximately  1  second 
anti  an  improvement  of  about  1 1  %  in  accuracy.  There  was  no  consistent  difference 
demonstrated  between  the  Long  and  the  Concept  line  drawings.  Because  the  Long 
Reduction  Method  can  be  completely  automated,  it  is  a  more  desirable  method  of 
constructing  line  drawings  for  the  LMC  than  the  manually  based  Concept  line 
drawing. 

The  RTs  for  this  experiment  were  1  to  2  seconds  longer  than  the  times  in 
Experiment  1.  In  both,  however,  the  times  were  well  within  the  time  available  for 
targeting  in  a  high  speed,  low  level  pass.  There  is  no  obvious  explanation  for  the  time 
differences  in  the  two  experiments:  two  different  experimenters  conducted  the  testing, 
which  may  have  inadvertently  influenced  the  time  pressure  that  the  subjects 
perceived:  in  addition,  in  Experiment  2  the  pretesting  experience  of  drawing 
templates  and  describing  photographs  may  have  caused  the  subjects  in  that 
experiment  to  be  slower  and  more  thoughtful.  Overall  operator  accuracy  in 
Experiment  2  was  3%  higher  than  in  Experiment  1. 

Finally,  in  this  experiment  as  in  the  first  experiment,  the  gains  produced  by  using 
the  Long  and  Concept  line  drawings  for  pattern  matching  showed  mainly  in  the 
accuracy  and  RTs  for  verification  of  matches.  Recognition  of  false  alarms 
(mismatches)  did  not  improve  as  a  function  of  the  type  of  line  drawing.  Figure  12 
shows  the  speed  and  accuracy  differences  as  a  function  of  Reduction  Method. 
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FIGURE  12.  Speed  and  Accuracy  Differences  as  a  Function  of  Reduction  Method. 
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There  was  some  slight  evidence  that  experience  in  making  line  drawings  speeded 
up  RTs  for  pattern  recognition.  Manual  tracing  of  the  photographs  to  make  a  line 
drawing  of  the  scene  stressed  the  visual,  as  opposed  to  the  verbal,  aspects  of  the 
pattern  matching  process.  Atwood  has  suggested  that  pattern  matching  is  primarily  a 
visual  task,  and  that  verbalization  concerning  the  patterns  to  be  matched  is  in  effect  a 
encoding  of  a  visual  process  into  a  verbal-auditor\'  proce.s.s  and  will  thus  take  time 
and  increase  RTs  (Reference  6).  Our  findings  support  this  interpretation.  However  in 
the  pretraining  that  was  used  in  this  experiment,  no  steps  were  taken  to  prevent 
subjects  from  verbalizing  about  the  photographs  while  simultaneously  tracing  them. 
Thus  our  findings  do  not  constitute  unequivocal  .support  for  this  interpretation. 

Error  analyses  showed  that  errors  were  greater  for  some  of  the  72  images  than  for 
others.  There  was  no  readily  apparent  interpretation  for  these  differences.  Neither  the 
type  of  scenery  nor  the  photographic  image  quality  appeared  to  account  for  the  errors. 

Finally,  the  traced  images  produced  in  the  pretraining  w'ere  qualitatively 
analyzed.  There  was  remarkable  consistency  between  subjects  in  the  lines  that  were 
included  in  the  tracings,  such  that  a  substantial  number  of  the  lines  were  included  by 
all  subjects.  In  addition,  each  subject  usually  included  at  least  one  line  that  was 
unique.  In  general,  the  practice  drawings,  when  digitized,  included  more  pixels  than 
tho.se  u.sed  in  the  experiments,  and  subjectively  at  least,  they  appeared  to  make  more 
coherent,  better  organized  scene  representations  than  the  Long  or  Concept  line 
drawings  u.sed  in  Experiment  2.  Intuitively  the.se  drawings  could  be  more  accurately 
used  for  scene  recognition  than  the  drawings  made  by  the  LMC.  w'hich  were  used  .so 
successfully  for  pattern  matching. 


CORRELATOR  MATCH  CRITERION 


In  the  LMC,  line  drawings  will  be  correlated  to  edge-enhanced  sensor  imagerv’. 
The  correlation  that  is  computed  is  the  sum  of  the  gray  scale  value  of  the  pixels  in  the 
edge-enhanced  sensor  scene  that  lie  under  the  lines  of  the  superimposed  line  drawing. 
As  the  view  seen  from  the  aircraft  sensor  approaches  that  generated  from  the  model 
(such  that  the  range  and  viewing  angle  are  the  same  as  the  one  used  to  generate  the 
line  drawing  from  the  model),  the  correlation  value  should  increase.  This  .section 
addresses  the  question:  What  should  the  magnitude  of  the  correlation  be  for  the  LMC 
to  call  out  a  match  between  the  line  drawing  and  the  .sensor  imagers  ? 

Our  general  strategy  to  address  this  que.stion  was  to  calculate  the  correlations 
between  each  of  our  72  pictures  and  their  matching  line  drawings  (using  the  Long 
drawings  from  Experiment  2),  and  between  each  picture  and  several  mismatched  line 
drawings.  But  becau.se  our  pictures  were  of  .several  different  sizes  and  had  different 
levels  of  contrast  from  one  another,  and  because  each  of  our  templates  contained 
different  numbers  of  pixels,  we  normalized  each  one  with  respect  to  gray  scale, 
picture  size,  and  template  size,  before  calculation  of  the  correlations.  We  then  plotted 
the  distribution  of  correlations  for  both  the  matches  and  the  mismatches.  Figure  13 
shows  the  cumulative  percentages  of  the  matches  and  the  cumulative  percentages  of 
the  mismatches.  If  a  correlation  .score  of  37  was  .set  as  a  criterion  for  the  LMC.  such 
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that  at  or  above  37  the  LMC  would  call  out  a  match,  then  from  Figure  13  we  can  see 
that  the  LMC  would  have  named  about  5%  of  mismatches  as  matches  (false  alarms), 
and  would  have  omitted  calling  out  about  7%  of  the  real  matches  (incorrect 
rejections).  In  terms  of  actual  frequencies  for  our  sample,  with  a  criterion  of  37,  the 
LMC  would  have  had  3  false  alarms  and  65  correct  recognitions  (hits).  The  LMC 
would  have  had  5  misses  (incorrect  rejections)  and  61  correct  rejections,  in  the  actual 
system,  operators  could  correct  the  false  alarms,  but  the  way  the  LMC  is  currently 
envisioned,  the  operators  would  presumably  not  see  or  correct  the  LMC  misses. 
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FIGURE  13.  Cumulative  Frequency  Distribution  of  the  Matches  and  the 
Inverse  of  the  Cumulative  Frequency  Distribution  of  the  Mismatches. 


In  an  actual  sy.stem,  the  criterion  could  be  adjusted  to  control  either  the  false 
alarm  rate,  the  hit  rate,  or  the  total  error  rate.  In  this  example  the  total  error  rate  was 
about  12%  (59c  false  alarms  and  7%  misses);  however  if  it  was  necessaiw  to  control 
the  miss  rate  to  less  than  5%,  the  criterion  could  be  set  at  25.  The  false  alarm  rate, 
however,  would  be  expected  to  rise  to  about  40%,  for  an  overall  error  rate  of  45%. 
This  example  illustrates  the  trade-offs  that  the  system  designers  will  need  to  consider 
when  selecting  the  correlation  score  to  be  used  as  a  criterion. 

We  recommend  a  similar  approach  to  set  the  actual  correlation  criterion  for  the 
LMC.  To  establish  a  suitable  criterion  and  obtain  estimates  of  false  alarms  and 
misses,  additional  testing  will  need  to  occur,  using  a  variety  of  scenes  from  actual 
.sensor  imagery,  along  with  line  drawings  that  are  made  from  appropriate  .source 
materials.  By  obtaining  performance  measurements  of  operators  using  the  LMC. 
system  level  estimations  of  hits,  correct  rejections,  false  alarms,  and  mis.ses  can  be 
made.  Adjustments  of  the  criterion  correlation  scores  can  be  used  to  change  these 
sy.stem  level  parameters,  or  if  acceptable  oerformance  levels  cannot  be  reached  by 
adjusting  the  criterion,  (hen  one  can  alter  the  number  of  pixels  in  the  line  drawings  to 
improve  the  operators'  ability  to  recognize  false  alarms. 
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Appendix 

VERBAL  DESCRIPTIONS  OF  EACH  PHOTOGRAPH  AND  PIXEL  COUNT 

FOR  THE 

CONCEPT  REDUCTION  METHOD 
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Verbal  Descriptions  of  Four  Judges,  and  pixel  counts. 

1.  Oil  tanks  on  hills  by  shore.  1500 

2.  Coastline  and  bay.  712 

3.  End  of  a  mountain  glacier.  1775 

6.  Lake  below  snowy  mountain.  1291 

7.  Mayan  ruins  in  the  forest.  2382 

8.  Mosque  and  park.  2618 

10.  Sand  dunes.  1698 

11.  City  freeway  (vertical).  2436 

13.  Overlook  of  an  inhabited  area.  2522 

14.  Coastal  road  with  two  headlands  (promontories).  880 

16.  Single  rock  arch.  2102 

17.  Bridge  across  river  by  oil  tanks.  1293 

18.  City  with  three  main  streets  (two  cross)  2668. 

23.  House  in  the  rocks  with  pool  on  shore.  1895 

29.  Desert  scene  with  distant  mountains.  2440 

30.  H  shaped  building.  863 

31.  Conical  hill  in  desert.  2385 

32.  Serpentine  river  in  mountains.  2229 

34.  Canyon  surrounded  by  four  buttes.  1594 

35.  Two  pinnacles  in  desert.  2471 

36.  Two  mountains  behind  a  lake  with  trees.  1718 

38.  Row  houses  in  a  city.  2477 

39.  Mountain  lake  with  curved  rocky  grassy  shore.  2030 

41.  Tower  and  city.  1809 

42.  City  with  bridges  over  river  for]cs.  2535 

44.  Stadium  with  skyscrapers  and  freeway.  2471 

45.  Large  arch  by  the  river.  1113 
47.  Road  through  woodland  hills  2278 
49.  targe  waterfall  by  a  city.  2473 

52.  Aereal  view  of  city,  skyscrapers,  cloverleaf.  2653 

53.  Reef  with  sandy  shoreline.  2182 

54.  Rivers  joining  with  a  bridge.  1825 

56.  Canyon  stream  seen  from  a  large  flat  rock.  1972 
58.  Horizontal  canal  through  rolling  farmland  2294 
62.  Loading  facility.  2195 
65.  star  shaped  fort  on  wooded  shore.  1585 

68.  Snow  capped  distant  peak  seen  from  shoreline  with 
trees.  1694 

69.  Rocky  desert  hills,  dunes,  peaks.  2474 

70.  Glacier  flowing  around  a  mountain  with  trees  in 
foreground.  2302 

71.  Aerial  view  of  lake  surrounded  by  large  mountains. 
1964 

75.  Foreground  rocky  pinnacles  with  smooth  slopes  behind. 
2268 

76.  Tall  jagged  snow  capped  mountain  with  smaller  peak  to 
left.  1540 

77-  Twin  skyscrapers  in  a  city  by  a  river.  2088 
79.  Big  flat  rock  with  snowy  mountains  behind.  1985 
81.  River  bend  with  flat  highland  (mesa)  in  background. 
1685 

83.  Snow  covered  pyramid  peak  in  rocky  flatland  1446 
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84.  Cliff  dwellings.  2436 

86.  Urban  peninsula  in  bay  with  proninant  rock  dome.  2496 

87.  Plains  with  mountain  peaks  behind.  2301 

96.  Reedy  mountain  lake  with  trees.  2470 

97.  Two  rock  arches.  1644 

100.  Rocky  shore  with  surf  and  palm  trees.  2034 
102.  Ocean  side  recreational  park  with  tower.  2515 

106.  Mountain  lake  created  by  a  dam.  2378 

107.  Canals  crossed  by  bridges  and  highway.  1995 

109.  A  lake  surrounded  by  sloped  hills  with  sparse  trees. 
2319 

110.  View  across  a  bay  of  a  city  at  night.  1802 

111.  Shoreline  houses  on  pilings.  2407 

114.  Ship  at  dock  with  bridge  in  background.  2159 

115.  Aereal  view  of  rough  coastline  and  coastal  city.  2638 

116.  Open  stadium  surrounded  by  parked  cars.  2544. 

118.  Two  barren  buttes.  2291 

120.  Skinny  arch  with  rocks  below.  2023 

121.  Large  butte  rising  from  field.  2197 

122.  Rugged  coast  with  roc]C6.  1689 

123.  Large  mountains  with  forked  river  below.  1558 
125.  Rocky  sloping  barren  hills.  2239 

127.  straited  dome  with  trees  below.  2441 

129.  Background  cliff  with  grassy  sand  hills.  2072 

130.  Sno%ry  stream  with  snowy  peak  behind.  589. 

141.  Towering  rocks  with  a  house  on  top.  2079 

142.  Crater  with  buildings  on  the  edge.  2061 

144.  Curved  river  with  trees  and  mountain  range  behind. 
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